1. Field of the Invention
This invention generally relates to multi processor computer systems, and more specifically, to methods and systems for mapping connections to CPUs in I/O Adapters in multi processor computer systems.
2. Background Art
Multi processor computer systems have become commonplace in the last few years, enabling multiple concurrent processes in systems to achieve higher speed networking. These multi processor systems execute multiple application threads, which interact with remote systems, when each thread may send and/or transmit data over one or more network connections. Some part of the network processing is performed by the operating system in execution context of these application threads, while other part of the processing is performed in context of the OS kernel (more specifically, network receive processing context), not associated with any particular thread context. When the same connection is handled by different CPUs (one executing network processing in the context of an application thread and one or more additional CPUs in kernel network receive context), the overhead of interaction between these processors (for controlling access to the shared connection state) may outweigh the benefits of using multiple CPUs. Thus, one of the more challenging aspects of utilizing multiple processors in scalable way is scheduling the application threads and in-kernel network processing across processors in a way that optimizes overhead for each processor individually and collectively.
To achieve scalability, a technology known as receive side scaling (RSS) may be used. This technology allows the network load from a network controller to be balanced across multiple processors, using multiple queues of received packets, when packets belonging to the same TCP connection are always placed by the adapter into the same queue. This allows scheduling of network receive processing on multiple CPUs in a controllable way, when packets arriving on the same connection are always handled by the same CPU. In addition, thread scheduling policy may then schedule an application thread which uses a particular connection on the CPU which handles the receive traffic of that connection, which ensures that all network processing for that connection is done on the same CPU, thus avoiding the overhead of interaction between different parts of the network processing.
RSS (Receive Side Scaling) adapters distribute packets to different receive queues, to allow processing in parallel on multiple CPUs, while keeping packet ordering within each connection. When a connection is opened (once per run), it is mapped into a receive queue (associated with a CPU), according to hash value of the connection tuple using a pre-defined hash function implemented by the adapter. During the adapter receive operations, the packets are directed to the proper receive queue (CPU).
Recently, Sun Microsystems introduced to the market a 10 Gigabit multi cores network processor that can implement different mappings. They can map packets based on packet class, ternary matches, or a pre-defined hash function (this is hardware based flow classification based on L1 to L4 packet header information). With these capabilities, a mapping policy of thread affinity has been proposed. This policy produces a one-to-one correlation of receive and transmit packets across the same TCP connection. While this mapping policy has some advantages, it does not maximize reuse of CPU cache or achieve maximum efficiency of TCP handling.